{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2019 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overview\n",
    "\n",
    "### Topic\n",
    "\n",
    "This notebook demonstrates using the AutoML API for Vision Classification.\n",
    "\n",
    "### Audience\n",
    "\n",
    "The audience for this notebook are software engineers (SWE) with limited experience in machine learning (ML).\n",
    "\n",
    "### Prerequistes\n",
    "\n",
    "One should be familar with:\n",
    "\n",
    "    - Python 3.X\n",
    "    - Google Cloud Platform (GCP) and using GCP buckets.\n",
    "    - Concept of Image Classification.\n",
    "    \n",
    "### Dataset\n",
    "\n",
    "This notebook using the Kaggle dataset for dog-breeds, located at:\n",
    "\n",
    "    https://www.kaggle.com/c/dog-breed-identification/data\n",
    "    \n",
    "From the Kaggle web page, you will need to select the Download All. Once downloaded to your laptop, you will need to unzip the folder. Once unzipped, there will be two additional zip files: train.zip and test.zip. For our purposes, it will be sufficient to unzip just the train.zip folder, which contains the training data.\n",
    "    \n",
    "This dataset contains train and test images for training an image classifier to classify 120 dog breeds. The train dataset contains approx. 10,0000 images.\n",
    "\n",
    "These images are not partitioned into folders based on class. Instead, all the images are in one folder and each has a unique label. The CSV file labels.csv has the mapping of each image ID to the label (dog breed). Note that the Id field in the CSV file is not a file path, but just the Id. Later, we will have to make a modified copy and replace the Id with the actual path to the file.\n",
    "\n",
    "### Objective\n",
    "\n",
    "The objective of this tutorial is to learn how to use the AutoML API to train a model for vision classification, deploy the model and do predictions using a gRPC or REST API interface.\n",
    "\n",
    "### Costs \n",
    "\n",
    "This tutorial uses billable components of AutoML Vision.\n",
    "\n",
    "Learn about [AutoML Vision Pricing](https://cloud.google.com/vision/automl/pricing)\n",
    "\n",
    "### Set up your local development environment\n",
    "\n",
    "**If you are using Colab or AI Platform Notebooks**, your environment already meets\n",
    "all the requirements to run this notebook. You can skip this step.\n",
    "\n",
    "**Otherwise**, make sure your environment meets this notebook's requirements.\n",
    "You need the following:\n",
    "\n",
    "* The Google Cloud SDK\n",
    "* The Google AutoML SDK\n",
    "* Git\n",
    "* Python 3\n",
    "* virtualenv\n",
    "* Jupyter notebook running in a virtual environment with Python 3\n",
    "\n",
    "The Google Cloud guide to [Setting up a Python development\n",
    "environment](https://cloud.google.com/python/setup) and the [Jupyter\n",
    "installation guide](https://jupyter.org/install) provide detailed instructions\n",
    "for meeting these requirements. The following steps provide a condensed set of\n",
    "instructions:\n",
    "\n",
    "1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)\n",
    "\n",
    "2. [Install Python 3.](https://cloud.google.com/python/setup#installing_python)\n",
    "\n",
    "3. [Install AutoML SDK](https://pypi.org/project/google-cloud-automl/) using the `pip install google-cloud-automl` command in a shell.\n",
    "4. [Install\n",
    "   virtualenv](https://cloud.google.com/python/setup#installing_and_using_virtualenv)\n",
    "   and create a virtual environment that uses Python 3.\n",
    "\n",
    "5. Activate that environment and run `pip install jupyter` in a shell to install\n",
    "   Jupyter.\n",
    "\n",
    "6. Run `jupyter notebook` in a shell to launch Jupyter.\n",
    "\n",
    "7. Open this notebook in the Jupyter Notebook Dashboard.\n",
    "\n",
    "### Set up your GCP project\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "1. [Select or create a GCP project.](https://console.cloud.google.com/cloud-resource-manager)\n",
    "\n",
    "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
    "\n",
    "3. [Enable the AI Platform APIs and Compute Engine APIs.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component)\n",
    "\n",
    "4. Enter your project ID in the cell below. Then run the  cell to make sure the\n",
    "Cloud SDK uses the right project for all the commands in this notebook.\n",
    "\n",
    "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands.\n",
    "\n",
    "Jupyter runs lines prefixed with `%` as automagic commands, which are interpreted within your IPython session. Automagic commands include `%ls`, `%pwd`, `%env` and `%pip` for example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROJECT_ID = \"[your-project-id]\" #@param {type:\"string\"}\n",
    "!gcloud config set project $PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authenticate your GCP account\n",
    "\n",
    "**If you are using AI Platform Notebooks**, your environment is already\n",
    "authenticated. Skip this step.\n",
    "\n",
    "**If you are using Colab**, run the cell below and follow the instructions\n",
    "when prompted to authenticate your account via oAuth.\n",
    "\n",
    "**Otherwise**, follow these steps:\n",
    "\n",
    "1. In the GCP Console, go to the [**Create service account key**\n",
    "   page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).\n",
    "\n",
    "2. From the **Service account** drop-down list, select **New service account**.\n",
    "\n",
    "3. In the **Service account name** field, enter a name.\n",
    "\n",
    "4. From the **Role** drop-down list, select\n",
    "   **Machine Learning Engine > AI Platform Admin** and\n",
    "   **Storage > Storage Object Admin**.\n",
    "\n",
    "5. Click *Create*. A JSON file that contains your key downloads to your\n",
    "local environment.\n",
    "\n",
    "6. Enter the path to your service account key as the\n",
    "`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "# If you are running this notebook in Colab, run this cell and follow the\n",
    "# instructions to authenticate your GCP account. This provides access to your\n",
    "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
    "# requests.\n",
    "\n",
    "if 'google.colab' in sys.modules:\n",
    "  from google.colab import auth as google_auth\n",
    "  google_auth.authenticate_user()\n",
    "\n",
    "# If you are running this notebook locally, replace the string below with the\n",
    "# path to your service account key and run this cell to authenticate your GCP\n",
    "# account.\n",
    "else:\n",
    "  %env GOOGLE_APPLICATION_CREDENTIALS your_path_to_credentials.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set Service Account Role for AutoML\n",
    "\n",
    "Add yourself and your service account to the AutoML Editor IAM role.\n",
    "\n",
    "    - Replace your-userid@your-domain with your user account.\n",
    "    - Replace service-account-name with the name of your new service account, for example service-account1@myproject.iam.gserviceaccount.com."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gcloud auth login\n",
    "!gcloud projects add-iam-policy-binding $PROJECT_ID \\\n",
    "   --member=\"[user:your-userid@your-domain]\" \\\n",
    "   --role=\"roles/automl.admin\"\n",
    "!gcloud projects add-iam-policy-binding $PROJECT_ID \\\n",
    "   --member=\"[serviceAccount:service-account-name]\" \\\n",
    "   --role=\"roles/automl.editor\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Allow AutoML service to access for Google Cloud project\n",
    "\n",
    "Allow the AutoML service account (custom-vision@appspot.gserviceaccount.com) to access your Google Cloud project resources:\n",
    "\n",
    "*This is a pre-existing global AutoML Vision service account which is separate from the project service account you just created; it is not visible in your project's list of service accounts.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gcloud projects add-iam-policy-binding $PROJECT_ID \\\n",
    "  --member=\"serviceAccount:custom-vision@appspot.gserviceaccount.com\" \\\n",
    "  --role=\"roles/ml.admin\"\n",
    "!gcloud projects add-iam-policy-binding $PROJECT_ID \\\n",
    "  --member=\"serviceAccount:custom-vision@appspot.gserviceaccount.com\" \\\n",
    "  --role=\"roles/storage.admin\"\n",
    "!gcloud projects add-iam-policy-binding $PROJECT_ID \\\n",
    "  --member=\"serviceAccount:custom-vision@appspot.gserviceaccount.com\" \\\n",
    "  --role=\"roles/serviceusage.serviceUsageAdmin\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a Cloud Storage bucket\n",
    "\n",
    "**The following steps are required, regardless of your notebook environment.**\n",
    "\n",
    "When you submit a training job using the AutoML Vision SDK, you must store the training\n",
    "data in a GCS bucket\n",
    "\n",
    "Set the name of your Cloud Storage bucket below. It must be unique across all\n",
    "Cloud Storage buckets. \n",
    "\n",
    "You may also change the `COMPUTE_REGION` variable, which is used for operations\n",
    "throughout the rest of this notebook. Make sure to [choose a region where Cloud\n",
    "AI Platform services are\n",
    "available](https://cloud.google.com/ml-engine/docs/tensorflow/regions). You may\n",
    "not use a Multi-Regional Storage bucket for training with AI Platform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BUCKET_NAME = PROJECT_ID + \"-vcm\"        #@param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Default compute region for AutoML\n",
    "COMPUTE_REGION='us-central1'\n",
    "\n",
    "! gsutil mb -l $COMPUTE_REGION gs://$BUCKET_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, validate access to your Cloud Storage bucket by examining its contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! gsutil ls -al gs://$BUCKET_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PIP Install Packages and dependencies\n",
    "\n",
    "Install addional dependencies not install in Notebook environment\n",
    "(e.g. XGBoost, adanet, tf-hub)\n",
    "\n",
    "- Use the latest major GA version of the framework."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -U google-cloud-storage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import libraries and define constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "# import the Google AutoML client library\n",
    "from google.cloud import automl_v1beta1 as automl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Copy the sample images into your bucket\n",
    "\n",
    "Copy the dog-breedf dataset into to your own bucket.\n",
    "\n",
    "This may take a couple of minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set this variable to the location of the training data on your laptop.\n",
    "DATASET=\"[my_path/dog-breed-identification]\"\n",
    "!gsutil -m cp -R [] $DATASET/train gs://$BUCKET_NAME/dog_breeds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the CSV file\n",
    "\n",
    "The sample dataset contains a CSV file with all of the image locations and the labels for each image. You'll use that to create your own CSV file:\n",
    "\n",
    "    1. Update the CSV file to point to the files in your own bucket.\n",
    "    2. Copy the CSV file into your bucket.\n",
    "    \n",
    "To learn more about preparing your training data, see [Preparing Training Data](https://cloud.google.com/vision/automl/docs/prepare)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat $DATASET/labels.csv | sed \"s:^:gs\\://$BUCKET_NAME/dog_breeds/:\" | sed \"s:,:\\.jpg,:\" > $DATASET/new_labels.csv\n",
    "!gsutil cp $DATASET/new_labels.csv gs://$BUCKET_NAME/csv/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create and Configure an AutoML client instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an AutoML client\n",
    "client = automl.AutoMlClient()\n",
    "\n",
    "# Derive the full GCP path to the project\n",
    "project_location = client.location_path(PROJECT_ID, COMPUTE_REGION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a dataset\n",
    "\n",
    "A dataset contains representative samples of the type of content you want to classify, labeled with the category labels you want your custom model to use. The dataset serves as the input for training a model.\n",
    "\n",
    "The main steps for building a dataset are:\n",
    "\n",
    "    - Specify a name for the dataset.\n",
    "    - Create a dataset and specify whether to allow multiple labels on each item.\n",
    "    - Import data items into the dataset.\n",
    "    \n",
    "The first step in creating a custom model is to create an empty dataset that will eventually hold the training data for the model. When you create a dataset, you specify the type of classification you want your custom model to perform:\n",
    "\n",
    "    MULTICLASS assigns a single label to each classified document\n",
    "    MULTILABEL allows a document to be assigned multiple labels\n",
    "    \n",
    "*In this tutorial, we are doing a MULTICLASS dataset*\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify a name for the dataset\n",
    "DATASET_NAME=\"[my-dataset-name]\"\n",
    "\n",
    "# Specify the image classification type for the dataset.\n",
    "dataset_metadata = {\"classification_type\": 'MULTICLASS'}\n",
    "# Set dataset name and metadata of the dataset.\n",
    "my_dataset = {\n",
    "    \"display_name\": DATASET_NAME,\n",
    "    \"image_classification_dataset_metadata\": dataset_metadata,\n",
    "}\n",
    "\n",
    "# Create a dataset with the dataset metadata in the region.\n",
    "response = client.create_dataset(project_location, my_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display response for creating an empty dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display the dataset information.\n",
    "print(\"Dataset name: {}\".format(response.name))\n",
    "print(\"Dataset id: {}\".format(response.name.split(\"/\")[-1]))\n",
    "print(\"Dataset display name: {}\".format(response.display_name))\n",
    "print(\"Image classification dataset metadata:\")\n",
    "print(\"\\t{}\".format(response.image_classification_dataset_metadata))\n",
    "print(\"Dataset example count: {}\".format(response.example_count))\n",
    "\n",
    "# Save the dataset ID\n",
    "dataset_id = response.name.split(\"/\")[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing items into a dataset\n",
    "\n",
    "After you have created a dataset, you can import item URIs and labels for items from a CSV file stored in a Google Cloud Storage bucket. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the full path of the dataset.\n",
    "dataset_full_id = client.dataset_path(\n",
    "    PROJECT_ID, COMPUTE_REGION, dataset_id\n",
    ")\n",
    "\n",
    "# Specify the location of the CSV file for the dataset\n",
    "CSV_DATASET = \"gs://\" + BUCKET_NAME + \"/csv/new_labels.csv\"\n",
    "input_config = {\"gcs_source\": {\"input_uris\": [CSV_DATASET]}}\n",
    "\n",
    "# Import data from the input URI.\n",
    "response = client.import_data(dataset_full_id, input_config)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display response from initiating the import of images into the dataset. **Call will return when import has completed. This may take upto 20 minutes**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# synchronous check of operation status.\n",
    "print(\"Data imported. {}\".format(response.result()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing Datasets\n",
    "\n",
    "A project can have multiple datasets, each used to train a separate model. You can get a list of the available datasets and can delete datasets you no longer need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = client.list_datasets(project_location, None)\n",
    "\n",
    "print(\"List of datasets:\")\n",
    "for dataset in response:\n",
    "    # Display the dataset information.\n",
    "    print(\"Dataset name: {}\".format(dataset.name))\n",
    "    print(\"Dataset id: {}\".format(dataset.name.split(\"/\")[-1]))\n",
    "    print(\"Dataset display name: {}\".format(dataset.display_name))\n",
    "    print(\"Image classification dataset metadata:\")\n",
    "    print(\"\\t{}\".format(dataset.image_classification_dataset_metadata))\n",
    "    print(\"Dataset example count: {}\\n\".format(dataset.example_count))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training a Cloud-Hosted Model\n",
    "\n",
    "You create a custom model by training it using a prepared dataset. AutoML API uses the items from the dataset to train the model, test it, and evaluate its performance. You review the results, adjust the training dataset as needed, and train a new model using the improved dataset.\n",
    "\n",
    "Training a model can take several hours to complete. The AutoML API enables you to check the status of training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify a name for your model.\n",
    "MODEL_NAME=\"[your-model-name]\"\n",
    "\n",
    "# Set training for a maximum of 1 hour\n",
    "train_budget=1\n",
    "\n",
    "# Set model name and model metadata for the image dataset.\n",
    "my_model = {\n",
    "    \"display_name\": MODEL_NAME,\n",
    "    \"dataset_id\": dataset_id,\n",
    "    \"image_classification_model_metadata\": {\"train_budget\": train_budget}\n",
    "}\n",
    "\n",
    "# Create a model with the model metadata in the region.\n",
    "response = client.create_model(project_location, my_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display response from initiating the training of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Training operation name: {}\".format(response.operation.name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display response from initiating the training of the model. **Call will return when training has completed. This may take upto 1 hour**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# synchronous check of operation status.\n",
    "print(\"Training done. {}\".format(response.result()))\n",
    "\n",
    "# Save the model ID\n",
    "model_id = response.result().name.split(\"/\")[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting information about a model\n",
    "\n",
    "After training your custom model is complete, you can get information about the newly created model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.cloud.automl_v1beta1 import enums\n",
    "\n",
    "# Get the full path of the model.\n",
    "model_full_id = client.model_path(PROJECT_ID, COMPUTE_REGION, model_id)\n",
    "\n",
    "# Get complete detail of the model.\n",
    "model = client.get_model(model_full_id)\n",
    "\n",
    "# Retrieve deployment state.\n",
    "if model.deployment_state == enums.Model.DeploymentState.DEPLOYED:\n",
    "    deployment_state = \"deployed\"\n",
    "else:\n",
    "    deployment_state = \"undeployed\"\n",
    "\n",
    "# Display the model information.\n",
    "print(\"Model name: {}\".format(model.name))\n",
    "print(\"Model id: {}\".format(model.name.split(\"/\")[-1]))\n",
    "print(\"Model display name: {}\".format(model.display_name))\n",
    "print(\"Image classification model metadata:\")\n",
    "print(\n",
    "    \"Training budget: {}\".format(\n",
    "        model.image_classification_model_metadata.train_budget\n",
    "    )\n",
    ")\n",
    "print(\n",
    "    \"Training cost: {}\".format(\n",
    "        model.image_classification_model_metadata.train_cost\n",
    "    )\n",
    ")\n",
    "print(\n",
    "    \"Stop reason: {}\".format(\n",
    "        model.image_classification_model_metadata.stop_reason\n",
    "    )\n",
    ")\n",
    "print(\n",
    "    \"Base model id: {}\".format(\n",
    "        model.image_classification_model_metadata.base_model_id\n",
    "    )\n",
    ")\n",
    "print(\"Model deployment state: {}\".format(deployment_state))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating the Model\n",
    "\n",
    "After training a model, AutoML Vision uses items from the TEST set to evaluate the quality and accuracy of the new model. For more information on how to interpret the evaluation, see [Evaluating models](https://cloud.google.com/vision/automl/docs/evaluate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the full path of the model.\n",
    "model_full_id = client.model_path(PROJECT_ID, COMPUTE_REGION, model_id)\n",
    "\n",
    "# List all the model evaluations in the model by applying filter.\n",
    "response = client.list_model_evaluations(model_full_id, None)\n",
    "\n",
    "print(\"List of model evaluations:\")\n",
    "for element in response:\n",
    "    print(element)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7rc2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
